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Today’s Topics 


Validity (a very mini-review) 

• National SAT Validity Study 

• Upcoming National Research - Placement Validity 
Study 

• Local Validity Research (ACES) 

• Admission Validity Studies (and Advisement) 

• Placement Validity Studies 
Useful College Board Web Sites 

• Validity Handbook 

• Research homepage 
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Validity 


• Test validity is the extent to which 
inferences, conclusions, and decisions 
made on the basis of test scores are 
appropriate and meaningful. 

• The use of tests for specific purposes 
(college admission, course placement, 
promotion, services, etc.) first requires 
validation of the test. 
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Conducting Validity Research 


NATIONAL VALIDITY RESEARCH 

• Responsible test developers and publishers must be able to 
demonstrate that it is possible to use the sample of behaviors 
measured by a test to make valid inferences about an 
examinee's ability to perform tasks that represent the larger 
domain of interest. 

• (e.g. SAT predicts FYGPA; SAT is related to other academic 
measures, etc.) 

LOCAL VALIDITY RESEARCH 

• Institutions should also gather evidence that the admission or 
placement decisions being made are fair or that the variables, 
such as test scores, are valid for their intended purpose. 



The National SAT Validity Study 


The College Board continues to examine the validity 
of the SAT across a national sample of four-year 
institutions and students, with outcomes including 
FYGPA, course grades, retention to the 2 nd Year, 
retention to the 3 rd year, etc. 


Data Collected as of February 2009 

Fall 2006 entering cohort - 1 st year data; 2 nd year data 

Fall 2007 entering cohort - 1 st year data 
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SAT Validity Study results - snapshot 


• Admission Validity Study 



SAMPLE 

110 colleges participating in Validity 
Study (N = 196,364) 

• Schools provided first year performance data 
for Fall 2006 cohort through the Admitted 
Class Evaluation Service™ (ACES ™) portal 

Restrict sample to students who 
completed the New SAT, submitted 
self reported High School GPA, and 
had a valid FYGPA (N=1 51 ,316) 
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Admission Validity Results (1 of 2) 


• SAT Writing has the highest correlation with 
FYGPA among the three individual SAT sections 
(Adj. r= 0.51). 

• In fact, SAT Writing alone has the same correlation with FYGPA as does 
SAT Critical Reading and SAT Math taken together. 


• As expected, the best combination of predictors of 
FYGPA is HSGPA and SAT scores (Adj. r= 0.62), 
reinforcing the recommendation that colleges use 
both HSGPA and SAT scores to make the best 
predictions of student success. 
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Admission Validity Results (2 of 2) 


• The adjusted correlation of HSGPA and FYGPA is 0.54, 
which is only slightly higher than the multiple correlation of 
the SAT (CR, M, and W combined) with FYGPA (Adj. r = 
0.53). 

• The increment in predictive validity attributable to the SAT 
when HSGPA is taken into account is 0.08. 

• The increment in validity attributable to the Writing section 
over and above the CR and M sections is 0.02. When 
HSGPA is also considered, the increment in validity 
attributable to the Writing section is 0.01 . 
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Future National Research - Placement Validity 


Coming soon... 

• What is the relationship between performance 
on the SAT and grades in first-year 
mathematics and English courses? 


• What percentage of students would be correctly 
placed if the SAT was used for course 
placement in first-year mathematics and 
English courses? 
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Local Validity Research - ACES™ 


• The Admitted Class Evaluation Service™ (ACES) 
is a free online service that predicts how admitted 
students will perform at your institution generally, 
and how successful they can be in specific classes. 

• Admission Validity 

• Placement Validity 


By conducting an ACES admission validity study, 
you can also get useful information to aid in 
advisement at your institution. 
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How ACES works... 

(ACES - http://professionals.collegeboard.com/higher-ed/validity/aces) 


• You design your study for your institution (choose 
predictors, criterion, subgroups, etc.) 

• Upload the data file with student records (ID 
variables; HSGPA; FYGPA) 


• In 25-35 business days, you will receive an email 
telling you to login into the ACES system to 
retrieve your ACES report and data file (which 
has been matched to additional College Board 
variables). 
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Requesting an Admission Validity Study 


• A min. of 75 student records required for an 
admission study. 

• Can specify up to 5 additional predictors - either 
ACES-supplied data or from an institution (provided 
that 75+ students in your sample have that 
additional variable). 


ACES automatically breaks down results by gender, 
race/ethnicity, and first language spoken (provided 
that there are 75+ students in the sample n at least 
2 levels of the subgroup). 

• Can also specify 2 additional subgroups - either ACES-supplied (e.g. 
degree-level goal, ability rating in math), from your data (e.g. resident 
versus commuter), or a combination. 
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ACES Admission Validity Study Report 


• Provides information on the most useful 
predictors of success at an institution 

• Provides information allowing 
administrators to narrow the number of 
factors considered in the admission 
process without loss of predictive ability 

• Provides optimal equations for predicting 
the success of future students 

• Provide a list of the students at risk 
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ACES Admission Validity Report 


ACES 


Admission Validity 
Report for Sample 
One University 
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Identifying Students at Risk 
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To help you target 
retention efforts at 
Sample One University, 
the predicted First- Year 
GPA has been added to 
each student's 
record on the electronic 
file returned to you. A list 
of IDs for students 
possibly at risk for 
dropping out or 
transferring is provided 
in Appendix C. 
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Placement Validity Studies 


• It is also possible to use ACES to conduct 
a predictive placement validity study using 
scores from: 

• SAT- Critical Reading, Writing, and Math 

• SAT Subject Tests 

• ACCUPLACER 

to predict success in a college course. 
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Requesting a Placement Study 


• You do not have to submit an additional data file for a 
placement study if you have submitted a 
comprehensive file for an admission validity study. 

• You may analyze up to five separate courses in your 
placement validity study. 

• There must be at least 30 student records for a placement validity 
study (or more, depending on the number of predictors used and 
the type of placement study requested). 

• For each course, you will begin by specifying the 
tests and the predictors you are most interested in 
evaluating. 

• You may use a single predictor, or up to five predictors, for a single 
course. 
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ACES Placement Validity Study 


• The predictive validity study provides two 
probability tables for cut scores, among other 
information: 

• For a course grade of B or higher 

• For a course grade of C or higher 
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Example of ACES Placement Chart 


Cut Scores Associated with Predicted Probability of Success Criterion: 
Final Course Grade of C or Higher in Engl 00 Using SAT Scores 


Cut Scoros AtoodoM wtth PwActod ProtttbUiry of Soccots 
Crttorlon Rnol Court* Grodo of C w Nfcghor is EnglOO 
Using SAT Scoftt 



Here you can see 
that an SAT CR 
score of 556, for 
example, is 
associated with a 
75% probability of 
obtaining a C or 
Higher in ENG 100 


n 
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Chart with Correlations and % Correctly Placed 


Individual predictors examined (SAT CR alone, SAT W 
alone) 


Composite predictor examined (SAT CR and W together) 


Logistic Bi serial Conflations' cl Pie dictors with Success an the Criterion 
Criterion: Final Course Grade oil Cor Higher in Engl 00 
Using SAT Scores 

ftedictor V'ariablels) 

StudfSampto 

Complete Data Sample 

N 

Logistic Biserial 
Correlation'' 

% Coirectiy 
Placed 

N 

Logistic Biserial 
Correlation'' 

% Correctly 
Flaced 

Individual PVedictors. 

SAT Critical Reading 

i92 

0.18 

60 

192 

0.16 

69 

SAT Writing 

4 BE 

020 

70 

192 

029 

70 

Composite Predictors 

Model Number 1 

192 

0J7 

68 

192 

0.d7 

68 

Model Number 1 includes SAT Critical Reading and SATWTitrig 

+ The logistic biserial correlation is a measure of the strength of association. H is related to a biserial [nrrelation but has been muddied In b= consistent with logistic regression and 
adapted Id simile aid miilicle predictors. 
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ACES Validity Handbook 

(http://professionals.collegeboard.com/higher-ed/validity/aces/handbook) 


• It is designed to serve as a 
general reference for validity 
and includes information about 
validity beyond what is 
specifically applicable to ACES. 


• It includes specific information 
about the types of validity 
studies and their design that are 
available through the ACES 
system and helps interpret 
ACES study results. 


Validity 

ACCS*'* 


☆ 


Admission Vatdty Study 
Placemen: VatdCy Study 
Validity Handbook 
Test Vstdty 


Vafadty Evdence 
Glossary 

Which Study is Right for You 


About Admescn Vatdfy 
Studies 


About Placement Vatdty Studes 

Predctrve Placement VafcdCy 
Studes 

Concurrent Placement Vabdty 
Studes 

Existng Placement Program 
Evaiuaten 

Avodng Potentel Probems 
FAQ 
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The College Board Research Homepage 

(www.collegeboard.com/research ) 
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Thank You 


• Researchers are encouraged to freely express 
their professional judgment. Therefore, points of 
view or opinions stated in College Board 
presentations do not necessarily represent official 
College Board position or policy. 

• Questions? 

• Interested in partnering with the College Board on 
national validity research projects? 

• Contact Emily Shaw: eshaw@colleaeboard.org 
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